Metric Recording

Understanding how metric recording works in Diginsight Components to track database costs and identify expensive operations across your application

Metric recording provides automatic capture of database operation costs, enabling us to identify expensive operations and optimize resource usage across our application.

Following is an example diagrams for diginsight.query_cost metric:

Database Cost by Method (Total RU)
================================================================================

ReportService.GenerateAnnualReport  │██████████████████████████████████████████████████ 15,420 RU (12 ops)
UserService.GetUserProfile          │████████████████████████████████████               8,730 RU (1,234 ops)
AnalyticsService.GetCustomerTrends  │████████████████████████████                       6,234 RU (156 ops)
ProductService.SearchProducts       │███████████████████████                            4,567 RU (2,345 ops)
CustomerService.GetCustomerHistory  │██████████████████                                 3,234 RU (567 ops)
NotificationService.SendBulkEmail   │██████████████                                     2,345 RU (89 ops)
PaymentService.ProcessRefund        │██████████                                         1,789 RU (123 ops)
UserService.UpdateProfile           │████████                                           1,456 RU (2,345 ops)
ProductService.GetProductDetails    │██████                                             1,234 RU (3,456 ops)
OrderService.GetOrderStatus         │████                                                 890 RU (4,567 ops)
CustomerService.SearchCustomers     │███                                                  675 RU (1,789 ops)

Scale: Each █ ≈ 308 RU

where database cost for every method or every query can be put in evidence.

Using Tagging we can obtain database cost for every application, for every customer or as an example, for every specific report type or every application view.

Metric recording is strategic resource for cost management that enables you to:

• 🔍 Identify database cost for every query with detailed Request Unit (RU) consumption tracking
• 📊 Track costs by caller method to pinpoint expensive business operations
  
• 🏷️ Filter by arbitrary tags (eg. application, customer, report type) for targeted analysis
• 💰 Contain expensive executions through data-driven optimization

Table of Contents

• 📋 Overview
  • Core Concepts
  • Key Capabilities
• 🔍 How It Works
  • Automatic Detection
  • Tag Enrichment
  • Query Normalization
• 💡 Strategic Use Cases
  • Database Cost Analysis
• 🔧 Customization
  • Custom Filters
  • Custom Enrichers
• � Reference
  • Key Components
  • Metric Structure
  • Configuration Options
  • Best Practices

📋 Overview

Core Concepts

Metric Recording in Diginsight Components works by automatically monitoring OpenTelemetry activities and extracting performance metrics from database operations. When combined with observable database extensions like CosmosDbExtensions, it provides comprehensive cost tracking without requiring manual instrumentation.

The centerpiece of database cost tracking is the QueryCostMetricRecorder, which captures CosmosDB query costs as the diginsight.query_cost OpenTelemetry metric.

Key Capabilities

Automatic Cost Tracking

• Records Request Units (RU) consumption for every database operation
• No manual instrumentation required - works with existing Diginsight telemetry
• Captures costs at the individual query level with full context

Rich Context Information

• Method names and caller chain analysis
• Database and container information
• Application and environment tagging
• Custom business context through extensible enrichment

Intelligent Filtering

• Query normalization to reduce metric cardinality
• Caller filtering to focus on business operations
• Custom filtering logic for specific use cases
• Configurable tag inclusion/exclusion

🔍 How It Works

Automatic Detection

The metric recording system operates through a sophisticated activity listening mechanism:

1. Activity Monitoring: Listens to OpenTelemetry activities from database operations
2. Cost Extraction: Identifies activities with query_cost tags indicating CosmosDB operations
   
3. Context Analysis: Analyzes the call chain to identify business methods and entry points
4. Metric Recording: Records histogram data with enriched tags for analysis

// When this CosmosDB operation executes...
var response = await container.ReadItemObservableAsync<User>(userId, new PartitionKey("users"));

// The metric recorder automatically captures:
// - query_cost: 2.45 RU
// - method: "ReadItemObservableAsync"
// - caller1: "UserService.GetUserProfile"
// - application: "MyApp"
// - container: "users"
// - database: "myapp-prod"

Tag Enrichment

Every recorded metric includes comprehensive tagging for detailed analysis:

Standard Tags (Always Present):

• method: The immediate database operation method
• entrymethod: The top-level entry point method
  
• application: Application name from entry assembly
• container: CosmosDB container name (if available)
• database: CosmosDB database name (if available)

Configurable Tags:

• query: Normalized query text (for pattern analysis)
• caller1, caller2, etc.: Business logic methods in the call chain
• Custom tags through IMetricRecordingEnricher implementations

Query Normalization

Raw queries are normalized to prevent metric cardinality explosion while preserving semantic meaning:

-- Original query
SELECT * FROM c WHERE c.id = '123e4567-e89b-12d3-a456-426614174000' AND c.timestamp > '2023-01-01T10:30:00Z'

-- Normalized query  
SELECT * FROM c WHERE c.id = '{GUID}' AND c.timestamp > '{DATETIME}'

Caller Filtering: Focus metrics on business operations by excluding infrastructure code:

// Configuration to surface business operations
options.IgnoreQueryCallers = new[]
{
    "BaseRepository*",           // Skip generic repository methods
    "CosmosDbExtensions.*",      // Skip extension helpers
    "*Middleware*"               // Skip framework middleware
};

// Result: Metrics show business methods like:
// - UserService.GetUserProfile
// - OrderService.ProcessOrder
// Instead of infrastructure methods like:
// - BaseRepository.GetItems

💡 Strategic Use Cases

Database Cost Analysis

🎯 Primary Value Proposition: Identify and contain expensive database operations before they impact your budget.

Query Cost by Operation Type:

// Track costs by operation to identify expensive patterns
var costsByOperation = metrics
    .Where(m => m.MetricName == "diginsight.query_cost")
    .GroupBy(m => m.Tags["method"])
    .Select(g => new {
        Operation = g.Key,
        TotalCost = g.Sum(m => m.Value),
        AvgCost = g.Average(m => m.Value),
        Count = g.Count()
    })
    .OrderByDescending(x => x.TotalCost);

Cost by Business Function:

// Identify which business operations consume the most resources
var costsByBusinessFunction = metrics
    .Where(m => m.MetricName == "diginsight.query_cost" && m.Tags.ContainsKey("caller1"))
    .GroupBy(m => m.Tags["caller1"])
    .Select(g => new {
        BusinessFunction = g.Key,
        TotalCost = g.Sum(m => m.Value),
        OperationCount = g.Count(),
        AvgCostPerOperation = g.Average(m => m.Value)
    })
    .OrderByDescending(x => x.TotalCost);

// Results might show:
// - ReportService.GenerateAnnualReport: 15,420 RU total
// - UserService.GetUserProfile: 8,730 RU total  
// - OrderService.ProcessBulkOrder: 6,890 RU total

🔧 Customization

Custom Filters

Create Custom Metric Filters:

// Filter to exclude certain operations from metrics
public class CustomMetricFilter : IMetricRecordingFilter
{
    public bool ShouldRecord(Activity activity)
    {
        // Skip health check operations
        if (activity.GetCustomProperty("IsHealthCheck") is bool isHealthCheck && isHealthCheck)
            return false;
            
        // Skip operations with very low cost (noise reduction)
        if (activity.GetCustomProperty("query_cost") is double cost && cost < 1.0)
            return false;
            
        // Skip operations from specific methods
        var method = activity.OperationName;
        if (method?.Contains("Ping") == true || method?.Contains("Echo") == true)
            return false;
            
        return true;
    }
}

// Register the custom filter
services.AddSingleton<IMetricRecordingFilter, CustomMetricFilter>();

Custom Enrichers

Add Custom Business Context:

// Enricher to add tenant/customer information
public class TenantMetricEnricher : IMetricRecordingEnricher
{
    private readonly IHttpContextAccessor _httpContextAccessor;
    
    public TenantMetricEnricher(IHttpContextAccessor httpContextAccessor)
    {
        _httpContextAccessor = httpContextAccessor;
    }
    
    public IEnumerable<KeyValuePair<string, object?>> ExtractTags(Activity activity)
    {
        var httpContext = _httpContextAccessor.HttpContext;
        if (httpContext == null) yield break;
        
        // Extract tenant ID from header or claims
        if (httpContext.Request.Headers.TryGetValue("X-Tenant-Id", out var tenantId))
        {
            yield return new KeyValuePair<string, object?>("tenant", tenantId.ToString());
        }
        
        // Extract user ID from claims
        var userId = httpContext.User?.FindFirst("sub")?.Value;
        if (!string.IsNullOrEmpty(userId))
        {
            yield return new KeyValuePair<string, object?>("user", userId);
        }
        
        // Extract feature flag information
        if (activity.GetCustomProperty("FeatureFlags") is Dictionary<string, bool> features)
        {
            foreach (var feature in features.Where(f => f.Value))
            {
                yield return new KeyValuePair<string, object?>($"feature_{feature.Key}", true);
            }
        }
        
        // Extract request context
        var requestPath = httpContext.Request.Path.Value;
        if (requestPath?.StartsWith("/api/reports/") == true)
        {
            var reportType = ExtractReportTypeFromPath(requestPath);
            if (!string.IsNullOrEmpty(reportType))
            {
                yield return new KeyValuePair<string, object?>("report_type", reportType);
            }
        }
    }
    
    private string? ExtractReportTypeFromPath(string path)
    {
        // Extract report type from API path like "/api/reports/financial/annual"
        var segments = path.Split('/', StringSplitOptions.RemoveEmptyEntries);
        return segments.Length >= 4 ? $"{segments[2]}_{segments[3]}" : null;
    }
}

// Register the custom enricher
services.AddSingleton<IMetricRecordingEnricher, TenantMetricEnricher>();
services.AddHttpContextAccessor(); // Required for HTTP context access

Advanced Enrichment with Business Logic:

// Enricher that adds cost classification
public class CostClassificationEnricher : IMetricRecordingEnricher
{
    public IEnumerable<KeyValuePair<string, object?>> ExtractTags(Activity activity)
    {
        if (activity.GetCustomProperty("query_cost") is not double cost)
            yield break;
            
        // Classify operations by cost
        var costTier = cost switch
        {
            < 5.0 => "low",
            < 25.0 => "medium", 
            < 100.0 => "high",
            _ => "critical"
        };
        
        yield return new KeyValuePair<string, object?>("cost_tier", costTier);
        
        // Add efficiency rating based on operation type
        var method = activity.OperationName;
        if (method?.Contains("ReadItem") == true && cost > 10.0)
        {
            yield return new KeyValuePair<string, object?>("efficiency", "inefficient_read");
        }
        else if (method?.Contains("Query") == true && cost > 50.0)
        {
            yield return new KeyValuePair<string, object?>("efficiency", "expensive_query");
        }
    }
}

📚 Reference

Key Components

• QueryCostMetricRecorder: Core component that captures CosmosDB query costs
• CosmosDbExtensions: Observable database operations that generate cost telemetry
• IMetricRecordingFilter: Interface for custom filtering logic
• IMetricRecordingEnricher: Interface for custom tag enrichment

Metric Structure

Metric Name: diginsight.query_cost Type: Histogram Unit: Request Units (RU) Description: “CosmosDB query cost in Request Units”

Standard Tags: - method: Database operation method - entrymethod: Top-level entry point
- application: Application name - container: CosmosDB container (if available) - database: CosmosDB database (if available)

Configurable Tags: - query: Normalized query text - caller1, caller2, etc.: Business method callers - Custom tags via enrichers

Configuration Options

public class QueryCostMetricRecorderOptions
{
    public bool AddNormalizedQueryTag { get; set; } = false;
    public int AddQueryCallers { get; set; } = 0;
    public string[] IgnoreQueryCallers { get; set; } = Array.Empty<string>();
    public int NormalizedQueryMaxLen { get; set; } = 500;
}

Best Practices

✅ Do: - Use normalized queries to reduce cardinality - Configure caller filtering to surface business operations - Implement custom enrichers for business context - Monitor high-cost operations and trends - Set up alerts for cost anomalies

❌ Don’t: - Enable detailed query logging in production without limits - Include high-cardinality data in custom tags - Record metrics for health checks or internal operations - Ignore the performance impact of extensive enrichment

---

💡 Pro Tip: Start with basic configuration and gradually add enrichment based on your specific analysis needs. The power of metric recording lies in its ability to provide strategic insights into your database costs and help you make data-driven optimization decisions.